In order to debug the periodic glitches on PR horizontal accelerometers - that introduce a rotation of the F0 along ty and generate interferometer unlocks - a series of tests have been performed on Sa_PR control system during the last weeks:

• The sensing element of the accelerometers were re-centered.
• In order to identify possible communication problems on RapidIO bus, 'keep sample' has been enabled on coil driver DSP board code.
• In order to identify possible issues due to the distribution box, channels used by the in-air cables were swapped: horizontal accelerometers previously connected to channels 1,2,3 were moved to channels 4,5,6 and vertical accelerometers from #4,5 have been moved to #1,2.
• In order to identify possible issues due to the dedicated DSP boards, horizontal accelerometers have been connected to the DSP board previously connected to the vertical ones and viceversa. This operation has required a re-design of the controllers.
• In order to identify possible issues due to cabling, mechanical action on in-air cables was performed. Even though this test introduced similar glitches in the signals, they were longer respect to the ones causing the unlocks. Moreover similar actions on the accelerometer cabling of other towers produced similar results in ther respective signals.

Unfortunately all these were inconclusive. Even though the third test did not produce any change, we decided in any case to replace the distribution box this morning. Primary, secondary and feedback coils have been measured as well.

Yesterday we carried out the survey for cheking the position of some of the NCal reference plate points @NE, those measurable and labelled as NCal_XX_02 in the starting survey (see also eLog53883) .

Tab.1 reports the final result in VRS coordinates obtained for the points measured and the different stations performed.

Fig.1 and 2 show some photos of the survey.

During last 18 hours additional problems were detected on traffic between the two SRIO switches ('Retry' on interlink #2)

After monitornig data flow at PR payload control along the whole day, I found communication errors related to slot #1 processor that received few 'retry' from RapidIO switch #0 and processor in slot #4 that received few 'NoACK' from same switch. These errors could be explained by operations on any of PR payload processors (for example new download or heavy monitornig via ethernet) but could of course be an ageing effect on processors and/or switches. 'Retry' could be generated by busy receiver. 'NoACK' is more worring since associated to 8b/10b deconding errors. 

If problem persists we shall consider some processor and switch replacement/relocation

After yesterday cleanup of DSP programs by Valerio, the total mean traffic on RapidIO switches at NI payload control (ScNI) changed from 7.4 Gb/s to 6.8 Gb/s. Even if a the 600 Mb/s difference is quite big, the traffic il well within switches capabilities. The following table shows data traffic in Mb/s between the 8 processors involved in payload control. The traffic is handled by two RapidIO switches interconnected with 3 links.

Yesterday, we carried out the survey to determine the VRS coordinates of the center of the top flange of the vacuum tank housing the NE Tiltmeter.

The flange center, labeled TILTNE_center, was calculated by measuring 4 diametrical points at 90°. Tab.1 reports the VRS coordinates obtained, Fig.1 to 4 some photos of the measured points and Fig.5 the scheme of the survey.

To determine the Z coordinate of the tiltmeter, it should be considered that Tab1. reports the coordinates referred to the center of the sphere (SMR target), so the Z has an offset of +25 mm from the top plane of the flange.

We analyzed some data to evaluate both cause and impact of synchronization problems experienced recently at NE tower.

Digital correction signal arriving as input to NE suspension control system from LSC servers is oversampled by a factor 4 by the receiving processor. Signal, converted from single to double precision, is then transferred to a second processor where it is low-pass filtered, decimated by a factor 4, converted back to single precision and sent back to data acquisition (Sc_NE_MIR_LCS_CORR). At the same time the 40 kHz signal is upsampled to 320kHz and sent to digital to analog converters.

Knowing exactly the lowpass filter, it is possible to compute the difference between sent signal (LSC_NE_CORR) and signal actually received. In case of signal not arrived in due time, we simply use the previous sample (keep-sample mechanism in DSP code)

The first attached plot shows the difference between actual Sc_NE_MIR_LCS_CORR and what it should have been in case of correct receival (after 400us delay correction). In 10 minutes of data we counted about 40 missing samples. To be notice that in the same time period we had no problem at WE where the delay was of 500 us.

We check that all events in previous plot are single missing sample and zooming in we can see the impulse response of low pass filter (note that low pass filter runs at 40 kHz while this is the decimated output):We check that all events in previous plot are single missing sample and zooming in we can see the impulse response of low pass filter (note that low pass filter runs at 40 kHz while this is the decimated output).

The computed difference signal has the same effect of a noise added to correction. The amplitude of the noise is correlated to the amplitude of the input signal and its spectral content. To have an estimation of the impact we can compare pulses amplitude with rms value of correction signal that in this case was about 0.4 V. The DAC noise spectrum at high frequency is about 0.1 uV/sqrt(Hz) therefore these events could actually have a small impact on short time GW measurements since they are comparable with expected high frequency DAC noise. Nevertheless, overall duty-cycle loss is in the order of 1 part over 10’000.

To be noticed that problem of synchronization was fixed as described in original entry

Sa_SR crate that was partially damaged by the blackout has been replaced with a spare. Now DSP master board (serial #43147) is back in slot #1 and MCH previous configuration has been restored in order to properly propagate the timing signals to the other DSPs. However, probably due to minor mechanical incompatibilities in the MicroTCA connectors, p4 coil driver board (serial #42074), connected to F0, has to be moved from slot #5 to slot #6: DSP code has been changed accordingly.  All SR loops are now closed.

The failure of the electric system on last Wednesday has required a complete recovery of the control electronics of all suspensions. At the start of the intervention on Thursday morning, all crates were ON, expect for SR ones where the smart PDU was frozen in an unknown state that required a reconfiguration of the device. Three DSP boards were damaged and has to be replaced with spares: 

• Sa_SR master board (serial #43131), connected to bottom ring LVDTs, was unreachable on the network and has been replaced with DSP #43147;
• Sc_BS PSD board (serial #42110), connected to marionette transversal optical lever, was unreachable on the network and has been replaced with DSP #45270;
• Sa_PR board (serial #41026), connected to both F0 horizontal and vertical LVDTs, was malfunctioning generating corrupted signals, probably due to a damaged ADC or DAC. It has been replaced with spare with serial #41049. 

Moreover the SR top crate was partially damaged by the blackout: the slot #1 that hosts the master board, connected to both DAQ and timing box, was not working properly anymore. In particular MCH was not capable of powering any DSP inserted in that slot (in order to exclude problems on the MCH itself, bottom and top MCHs were temporarily exchanged). For this reason the spare LVDT board has to be installed on slot #7, action that required a reconfiguration of the MCH in order to properly propagate the timing signals to the other DSPs.

After the hardware interventions all crates of all towers has to be resynchronized with timing for proper DAQ transmission and all controls loops has to be verified and closed. The system was fully operational on Friday evening as reported by the on-shift operator (entry #60260).

Last May 4, we carried out the topographic survey of the height difference (N201-N200; W201-W200) between the tunnel and the tower basement within both End Buildings. This height difference is subjected to differential subsidence over the years and has to be periodically monitored.

Tab.1 reports the results for the NE building (N201-N200) and Tab.2 for the WE building (W201-W200).

Fig.1 and Fig.2 show for both cases the relevant graph of the trend over time and the curve fitting.

Last Friday May 5, we carried out the survey to determine the VRS coordinates of the reference point "1500W_ROBOT_00", marked on the floor and needed for the robot calibration (Fig.5).

Fig.1 to 5 show some pictures of the survey and Fig.6 the scheme of the points measured.

The result in VRS coordinates is reported in Tab.1.

During the analysis of the seismic noise associated to the heater cycle (INF_NEB_HEATER_TE) in ENV_NEB_SEIS_N we found that

• The noise in the seismograph disappears when the heater have been changed (from 27 jan 2023 to 05 Feb 2023). 
• A new noise appears in the velocimeters sensors array (NN_NEB_ACC_07,08,09) arond ~35 Hz and ~145 Hz. See HEATER_ACC_Z_0809_050218001830.png and NEB_ACC_08_08_1hour_5022023.png

These noises where related to:

• the ~145 Hz associated to the NEB_COMP1_PRES. See glitch145Hz_COMP1_PRES.png
• the ~35 Hz associated to the INF_NEB_AIR_PRES (now called INF_NEB_AIUR_COMP_LINE_PRES). See glitchNN_zoom.png

The first one (COMP1) is the compressor, the latter one (AIR_PRES) is a channel which follows the compressors 1 and 2. 

For further inforation see LogBook_Draft4.pdf

Yesterday we have completed the check of the position of the vacuum tube supports along the North Tunnel, for the part placed between @progr.1200m and @progr.1600m.

Tab.1 reports the results obtained as both local and VRS coordinates.

Fig.1 shows a detail of one of the points and Fig.2 the scheme of one of the station (St01).

In order to check the position of the vacuum tube axis in correspondence of the supports, consequently the several re-alignment operations carried out during past years, last Wed 11 Jan we carried out the survey of the position of some vacuum tube supports along the North Tunnel, placed between about @progr.1475m and @progr.1550m (partly inside North Tunnel and inside 1500N Building).

For convenience and speed of execution, the points taken as reference were placed on the upper metallic frame of the supports, whose position is at a constant deltaZ distance from the vacuum tube axis. Fig.1 reports a detail of one of the points and Fig.2 the scheme of the survey carried out.

Tab.1 reports the results obtained as both local and VRS coordinates.

After the installation @1500W Building of the floor supports and their tensioning (Fig.1), carried out for the future placement of the scientific equipments in the SQZ-R&D Lab, we made a check survey of the reference points located inside the SQZ-R&D Lab, in order to exclude any unwanted displacement.

The check has been carried out measuring, before and after the stiffening of the floor, the same reference points located inside and outside the Lab (Fig.2, 3 and 4) and surveyed by the same single station of AT403 laser tracker.

The first survey considered was that of framing the reference points in the VRS network, reported in eLog58222, while the second survey has been carried out last Wed, Jan 11.

Then, the difference of the Z coordinate between each reference point has been calculated, and the result is shown in Tab.1.

As reported, between the initial survey (Staz07) and after the installation of the structural reinforcements (Staz08), the differences of the deltaZs for each point located inside the Lab (1500W-06, 1500W-07 and 1500W-08) result less than 0.4 mm, thus within the accuracy of this kind of survey. Moreover, the differences are of the same order also for the all other points outside of the intervention area, therefore within the expected accuracy.

As conclusion, it can be stated that no raising of the height of the floor has been measured due to the floor stiffening.

Last Dec 6, we carried out the survey of the position of additional reference points located in the 1500W building and inside the SQZ-R&D Lab, required for future checks after the installation of the new apparatus inside the Lab.

Tab.1 reports the VRS coordinates obtained for the reference points measured, 1500W-01 to 1500W-05 in the building and 1500W-06 to 1500W-08 inside the Lab.

Fig.1 to 4 show some pictures of the survey.

The first ETRF position of the station VIRG00ITA (Fig.1) is now available on EUREF webpage https://www.epncb.oma.be/_networkdata/siteinfo4onestation.php?station=VIRG00ITA (see also previous eLog53935 and eLog55014).

The ETRF2000 coordinates calculated by EPN are the following (Tab.1), valid in the GPS calendar period 163/2022 - 211/2022, while the velocities are not availble yet.

The periodic monitoring campaing of the topographic heights of buildings and tunnels has been completed last week with the 14th survey of the link between the Central Building and the Mode Cleaner (see also related entries eLog 55893 and eLog 56068 for West and North Tunnel respectively).

Tab.1 reports the result of the survey and Fig.1 shows the CB-MC tunnel profile, containing graphically the trend of the subsidence over-time.

Last week we completed the 18th height monitoring campaign of the North Tunnel.

Tab.1 summarizes the result of the survey and Fig.1 shows the tunnel profile, that reports graphically the trend of the subsidence over time.

Last week we completed the 18th height monitoring campaign of the West Tunnel.

Tab.1 summarizes the result of the survey and Fig.1 shows the tunnel profile, that reports graphically the trend of the subsidence over time.

Two problems on MC suspension control arose during the shutdown: 

• Sa_MC MCH and DSP boards became unreachable on the network from the afternoon of April 13th: the cause was a a damaged Ethernet cable connected to the MCH.  Margarita replaced it yesterday. 
• Sc_MC DSP boards stopped transmitting data to DAQ: after a long investigation performed yesterday evening, with the help of Alain and Nicolas, the origin of the problem was identified: MCH router was damaged, disabling all data traffic on the RapidIO bus. This morning the device (IP 172.16.1.14) has been replaced with a new one (IP 172.16.1.25). Data are now back. MCH Tango server configuration has been updated.

Since last February 8, the GNSS Permanent Station VIRG00ITA (see also eLog 53935) is present in the list of "Proposed Stations" of the EUREF Permanent GNSS Network https://www.epncb.oma.be/_networkdata/proposed.php, an important step for the final insertion of VIRG00ITA in the European Network of GNSS permanent stations https://www.epncb.oma.be/.

The RinexV3 data are now available also through the dedicated station link https://www.epncb.oma.be/_networkdata/datacalendar.php?station=VIRG00ITA&year=2022&month=2&rv=3&c=epn